Implement workaround to allow SNO installations for OKD/FCOS #7445
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes the first time during installation. FCOS does not provide tools such as OpenShift Client (oc) or hyperkube which are used during single-node cluster installation at first boot (e.g. oc in bootkube.sh [0]). RHCOS and SCOS include these tools, but FCOS has to pivot the root fs [1] to okd-machine-os [2] first in order to make those tools available. Pivoting uses 'rpm-ostree rebase' but during SNO installation the node will be booted from a FCOS Live ISO where the root fs and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and necessary tools for SNO installation will not be available, causing the setup to stall. This patch works around this issue by mounting /usr and /etc as overlay filesystems which contains both the content of the live iso as well as from the okd-machine-os container image. [0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template [1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template [2] https://github.com/openshift/okd-machine-os
Contributor
|
LGTM! |
Contributor
|
Maybe add a comment in-code that this is supposed to be temporary, and ideally also add a link to an upstream issue. |
Contributor
|
/retest-required |
Contributor
|
/approve |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sadasu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
Contributor
Author
There is no upstream issue which is why i linked the RFE about allowing rpm-ostree rebase on live iso. I dare to write temporary because a proper solution is not easy and it might turn into something which is perma-temporary 😅 |
Contributor
|
/lgtm |
Contributor
Author
|
OpenShift's conformance test suite fails with: Seems to be unrelated to this change. Is there anything we can do except for |
Contributor
|
I'd like to see one of the OKD/FCOS E2Es go green. |
Contributor
Author
|
/retest-required |
Contributor
Author
|
/test okd-e2e-aws-ovn |
Contributor
Author
|
/test okd-e2e-aws-ovn-upgrade |
Contributor
Author
|
@LorbusChris one okd/fcos job is green now 😄 |
Contributor
Author
|
/label acknowledge-critical-fixes-only |
openshift-ci
bot
added
the
acknowledge-critical-fixes-only
Contributor
|
LGTM, I'm in favor of merging this now as it only affects OKD, and enables SNO installs for OKD 1.14. |
Contributor
|
@JM1: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Contributor
Author
|
/cherry-pick release-4.14 |
|
@JM1: new pull request created: #7479 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Contributor
Author
|
/cherry-pick release-4.13 |
|
@JM1: new pull request created: #7480 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
JM1
added a commit
to JM1/openshift-installer
that referenced
this pull request
Oct 27, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
JM1
added a commit
to JM1/openshift-installer
that referenced
this pull request
Nov 28, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
aleskandro
pushed a commit
to aleskandro/installer
that referenced
this pull request
Nov 30, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
JM1
added a commit
to JM1/openshift-installer
that referenced
this pull request
Nov 30, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
openshift-cherrypick-robot
pushed a commit
to openshift-cherrypick-robot/installer
that referenced
this pull request
Dec 12, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
JM1
added a commit
to JM1/openshift-installer
that referenced
this pull request
Jan 22, 2024
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
(cherry picked from commit b2bbc85)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes the first time during installation. FCOS does not provide tools such as OpenShift Client (
oc) or hyperkube which are used during single-node cluster installation at first boot (e.g.ocin bootkube.sh). RHCOS and SCOS include these tools, but FCOS has to pivot the root fs to okd-machine-os first in order to make those tools available.Pivoting uses
rpm-ostree rebasebut during SNO installation the node will be booted from a FCOS Live ISO where the root fs and/sysrootare mounted read-only. Thusrpm-ostree rebasefails and necessary tools for SNO installation will not be available, causing the setup to stall.This patch works around this issue by mounting
/usrand/etcas overlay filesystems which contains both the content of the live iso as well as from the okd-machine-os container image.